Surveillance with reanalysis of screening data

ABSTRACT

A surveillance system may include at least one controller configured to receive information data from at least one information source and to control operation of at least one controllable subsequent information source based, at least in part, on the information data. A surveillance method may include analyzing screening data, obtaining information data, and reanalyzing the screening data based, at least in part, on the obtained information data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a division of U.S. patent application Ser. No.11/740,133, filed Apr. 25, 2007 and entitled “Surveillance Systems andMethods with Subject-Related Screening”, which is a continuation-in-partof U.S. patent application Ser. No. 11/679,106, filed Feb. 26, 2007 andentitled “Multi-Source Surveillance System,” which is a continuation ofU.S. patent application Ser. No. 10/825,530, filed Apr. 14, 2004 andentitled “Multi-Source Surveillance System,” which issued as U.S. Pat.No. 7,205,926. Additionally, this application claims benefit under 35U.S.C. §119(e) to U.S. Provisional Patent Application Ser. No.60/795,434, filed Apr. 26, 2006 and entitled “Security Checkpoint;” U.S.Provisional Patent Application Ser. No. 60/801,961, filed May 19, 2006and entitled “Security Checkpoint;” U.S. Provisional Patent ApplicationSer. No. 60/810,598, filed Jun. 1, 2006 and entitled “SecurityCheckpoint;” and U.S. Provisional Patent Application Ser. No.60/846,590, filed Sep. 21, 2006 and entitled “Security Checkpoint.” Thisapplication further is related to U.S. patent application Ser. No.11/740,155 filed on Apr. 25, 2007 and entitled “Surveillance ofSubject-Associated Items with Identifiers,” and U.S. patent applicationSer. No. 11/740,193 filed on Apr. 25, 2007 and entitled “Surveillancewith Subject Screening.” The complete disclosures of the aboveapplications are herein incorporated by reference for all purposes.

BACKGROUND OF THE DISCLOSURE

Millimeter wave signals are used for radar and telecommunications. Theyare also capable of being used to produce an image of a subject bydirecting millimeter-wave signals at the subject and detecting thereflected signal. Examples of such imaging systems are described in U.S.Pat. Nos. 5,455,590; 5,557,283; 5,859,609; 6,507,309; 6,703,964;6,876,322; and 7,034,746; U.S. Patent Application Publication Numbers2004/0090359, 2004/0140924, 2006/0066469, and 2007/0075889; and U.S.patent application Ser. No. 11/088,470 filed Mar. 24, 2005. The completedisclosures of the above patent references are herein incorporated byreference for all purposes.

Imaging systems, including those systems described above, providerelatively detailed images of a subject, including a person's body andany objects carried by the person, whether or not the objects areconcealed. An operator can view images of a subject and visuallyidentify objects included with the person. Other detection devices canalso be used to obtain other information about a subject. For instance,the subject can pass through a metal detector to determine if thesubject has a detectable metal object.

Additionally, or alternatively, the surveillance system may includeimaging systems and/or other detection devices to obtain informationabout subject-associated items, such as divested objects, checkedbaggage, and carryon baggage. For example, imaging systems may be usedto produce images of checked baggage. An operator can view images ofchecked baggage and visually identify objects included with thatbaggage. Other detection devices can also be used to obtain otherinformation about subject-associated items. For example, trace detectiondevices may be used to determine if the subject-associated item has anexplosive substance or an illegal drug.

When the surveillance system is part of a process of boarding a publictransportation vehicle, or entering a public or protected facility,substantial delays may be realized. Further, the effectiveness of thesurveillance depends on the accuracy and attentiveness of the operatoror operators.

SUMMARY OF THE DISCLOSURE

Some embodiments provide a surveillance system. In some embodiments, thesurveillance system may include a first screening apparatus configuredto produce first screening data representative of screening at least aportion of a subject, the subject including a person and one or moreobjects with the person; at least one controllable information source ofsubject-associated item information data related to one or moresubject-associated items, wherein at least one of the subject-associateditems includes checked baggage; and at least one controller configuredto receive the first screening data and to control operation of the atleast one controllable information source of subject-associated iteminformation data based, at least in part, on the first screening data.

In some embodiments, the surveillance system may include a firstinformation source of checked-baggage information data related tochecked baggage associated with a subject; at least one controllableinformation source, other than the first information source, of at leastone of (a) subject information data related to the subject and (b)subject-associated item information data related to one or moresubject-associated items; and at least one controller configured toreceive the checked-baggage information data and to control operation ofthe at least one controllable information source, other than the firstinformation source, based, at least in part, on the checked-baggageinformation data.

In some embodiments, the surveillance system may include a firstscreening apparatus adapted to transmit toward and receive from asubject in a subject position, first electromagnetic radiation in afrequency range of about 100 MHz to about 2 THz, from positions spacedfrom the subject position, the subject including at least a portion of aperson in a subject position and detectable objects carried by theperson, the first screening apparatus producing from the receivedradiation, a first image signal representative of a first image of atleast a portion of the subject; a second screening apparatus differentthan the first screening apparatus, and adapted to detect a givencharacteristic of an object potentially carried by at least onesubject-associated item, and to produce a screening signalrepresentative of the detection of the given characteristic; and acontroller adapted to control operation of the first and secondscreening apparatus, and to produce, from the image signal, image datarepresentative of the image of the at least a portion of the subject andscreening data from the screening signal, to relate the image data tothe screening data, and to produce from the related image data andscreening data, relational information data about whether at least oneof (a) the person and (b) the at least one subject-associated item ispotentially carrying an object having the given characteristic.

Some embodiments provide a surveillance method. In some embodiments, thesurveillance method may include screening at least a portion of asubject-associated item associated with a subject; generating screeningdata based, at least in part, on the screening of at least a portion ofa subject-associated item; analyzing the screening data; obtaining,after analyzing the screening data, information data related to at leastone of (a) the subject and (b) one or more subject-associated itemsother than the screened subject-associated item; and reanalyzing thescreening data based, at least in part, on the information data, and ina manner that is at least partially different from the analyzing of thescreening data.

In some embodiments, the surveillance method may include screening atleast a portion of a subject, the subject including a person and one ormore objects with the person; generating screening data based, at leastin part, on the screening of at least a portion of a subject; analyzingthe screening data; obtaining, after analyzing the screening data,information data related to one or more subject-associated items; andreanalyzing the screening data based, at least in part, on theinformation data, and in a manner that is at least partially differentfrom the analyzing of the screening data.

In some embodiments, the surveillance method may include associating,with respective identifiers, a person and one or more carried itemsselected by the person to be carried by the person into acontrolled-access area; screening the one or more carried items whilephysically separate from the person; and matching the identifier of theone or more screened carried items with the identifier of the person,and, if the identifiers match, giving possession of the screened carrieditems to the person prior to the person entering the controlled-accessarea.

Some embodiments provide a surveillance portal. The surveillance portalmay include a first screening apparatus adapted to detect on a subjectin a subject position a first characteristic of an object potentiallycarried by at least one of (a) the subject and (b) one or moresubject-associated items, and to produce a first screening signalrepresentative of the detection of the first characteristic; a secondscreening apparatus that is different from the first screening apparatusand is adapted to detect on the subject in the subject position a secondcharacteristic of an object potentially carried by at least one of (a)the subject and (b) the one or more subject-associated items, and toproduce a second screening signal indicating the detection of the secondcharacteristic; and a controller adapted to control operation of thefirst and second screening apparatus, to produce first screening dataindicating the detection of the first characteristic from the firstscreening signal, and second screening data indicating the detection ofthe second characteristic from the second screening signal, to relatethe first screening data to the second screening data, and to producefrom the related first and second screening data, relational informationdata about whether at least one of (a) the subject and (b) the one ormore subject-associated items is potentially carrying an object havingat least one of the first and second characteristics.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram depicting an example of a general surveillancesystem having an imaging source and a supplemental subject informationsource.

FIG. 2 is a general flow chart illustrating an example of a method ofoperation of the surveillance system of FIG. 1.

FIG. 3 is an exemplary diagram illustrating various possibleconfigurations for subject information assemblies usable in thesurveillance system of FIG. 1, and having up to three sensors that moverelative to a subject.

FIG. 4 is a block diagram illustrating an example of a subjectinformation assembly of FIG. 3 having two sensor apparatus.

FIG. 5 is a general diagram showing an example of a surveillance systemaccording to FIG. 1 and including a subject information assembly of FIG.4.

FIG. 6 is a side view of another example of a subject informationassembly of FIG. 3.

FIG. 7 is a top view of yet another example of a subject informationassembly of FIG. 3.

FIG. 8 is an image generated by a millimeter-wave imaging system of amannequin having objects hidden by clothes on the mannequin.

FIG. 9 is an enlarged view of a portion of the image of FIG. 8 showingan object.

FIG. 10 is a graphic illustration of the values of a kernel used forobject detection.

FIGS. 11A and 11B illustrate a sequence of steps illustratingapplication of a kernel to an image.

FIG. 12 is an image resulting from applying the kernel of FIG. 10 to theimage of FIG. 8 to identify suspect areas of the image.

FIG. 13 is an image resulting from applying a threshold to the image ofFIG. 12.

FIG. 14 is an image resulting from dilating the highlighted areas of theimage of FIG. 13 and removing small regions.

FIG. 15 is an image resulting from applying bounding rectangles to thehighlighted areas of the image of FIG. 16.

FIG. 16 is an image resulting from dilating the highlighted areas of theimage of FIG. 17.

FIG. 17 is a copy of the original image of FIG. 8 superimposed with theoutlines of the highlighted areas of FIG. 16, which correspond withsuspect regions in the original image.

FIG. 18 is an exemplary display of a surveillance system having animaging apparatus and a zonal metal detector, identifying varioussuspect regions of a subject and zones of the subject where metal isdetected.

FIG. 19 is a display similar to FIG. 18 in which the subject images arecropped and a primary image is enlarged to show portions of the subjectimages having detected metal.

FIG. 20 is a block diagram depicting another example of a generalsurveillance system having a subject information source assembly and asubject-associated item information source assembly.

FIG. 21 is a general flow chart illustrating an example of a process ofboarding a vehicle for a subject and subject-associated items and anexample of how the surveillance system of FIG. 20 may be used with thatprocess.

FIG. 22 is a general flow chart illustrating an example of a method ofoperation of the surveillance system of FIG. 20.

FIG. 23 is a general flow chart illustrating another example of a methodof operation of the surveillance system of FIG. 20.

FIG. 24 is a general flow chart illustrating another example of a methodof operation of the surveillance system of FIG. 20.

FIG. 25 is a general flow chart illustrating another example of a methodof operation of the surveillance system of FIG. 20.

FIG. 26 is a general diagram showing an example of a system ofassociating a subject with carried items using the surveillance systemof FIG. 20.

FIG. 27 is a general diagram showing another example of a system ofassociating a subject with carried items using the surveillance systemof FIG. 20.

FIG. 28 is a general flow chart illustrating an example of a method ofassociating a subject with carried items using the surveillance systemof FIG. 20.

FIG. 29 is a general diagram showing an example of a subject informationassembly usable in the surveillance system of FIG. 20, and havingmultiple screening apparatus.

DETAILED DESCRIPTION OF THE DISCLOSURE

FIG. 1 depicts a general surveillance system 20 including a controller21 and a subject-information assembly 22 including a plurality ofsubject-related information sources 24. The subject information sourcesmay include one or a plurality of screening or sensor apparatus 26, suchas an imaging apparatus 28, and depending on the configuration, one or aplurality of supplemental subject information sources 30. The subjectinformation sources provide information relative to a subject 32 in asubject position 34.

A subject 32 includes all that is presented in an interrogation stationof a screening apparatus, whether human, animal, or inanimate object.For example, if a person 36 is in an interrogation station, the subject32 includes the person as well as any objects 38 with an/or supported onthe person, such as watches, keys, jewelry, pocket or other knives,coins, clothing accessories, guns, and/or any other objects.

An example of an imaging apparatus 28 is an active imaging apparatusadapted to transmit toward and receive from a subject in a subjectposition, millimeter-wave electromagnetic radiation. In some examples,the millimeter-wave radiation may be in the frequency range of 100Megahertz (MHz) to 2 Terahertz (THz) range. Generally, the correspondingwavelengths range from several centimeters to a few micrometers. Suchradiation may be either at a fixed frequency or over a range or set offrequencies using several modulation types, e.g. chirp, pseudorandomfrequency hop, pulsed, frequency modulated continuous wave (FMCW), orcontinuous wave (CW).

Some embodiments of imaging apparatus may use a transmitting signal thatincorporates frequencies in the range of 24 to 30 GHz, FMCW modulation,and having signal content that meets FCC unlicensed operationrequirements and is outside of any restricted US Government frequencybands. Pulse lengths may range from 2-10 microseconds. Antennae beamwidths may range from 20-120 degrees for broad beam implementation, orless than 30 degrees for narrow beam width applications, depending onthe image formation signal processor requirements. Various systempolarizations may be used. Examples include same polarization, crosspolarization, elliptical polarization, right circular polarization,and/or left circular polarization.

Certain natural and synthetic fibers may be transparent orsemi-transparent to radiation of such frequencies and wavelengths,permitting the detection and/or imaging of surfaces positioned beneathsuch materials. When the subject of interrogation is a clothedindividual, image information about portions of a person's body coveredby clothing can be obtained with system 20, as well as those portionsthat are not covered by clothing. Further, image information relative toobjects carried by a person beneath clothing, regardless of how they aresupported on the person, can be provided with system 20 for metal andnon-metal object compositions, such as those used for weapons andcontraband.

Controller 21 can be adapted to operate the sensor apparatus 26. In theexample shown in which a sensor apparatus is an imaging apparatus 28,the controller is adapted to produce image data representative of animage of the subject from received radiation. Supplemental subjectinformation sources 30 provide information about the subject 32 that isrelatable to objects 38 potentially carried by the person 36. The sourcemay be of a type that is different from imaging apparatus 28. Thecontroller then is adapted to produce relational information relatingthe produced image signal and the subject information.

In some surveillance systems 20, a supplemental source 30 of subjectinformation is a second sensor apparatus 26 adapted to detect a givencharacteristic of an object potentially carried by a person in thesubject position. The controller 21 in such a system 20 is accordinglyadapted to produce relational information about whether the person iscarrying an object having the given characteristic. The second sensorapparatus may be adapted to detect one or more of a variety of subjectinformation. The sensor apparatus may function sequentially orconcurrently with other sensor apparatus. For example, the second sensorapparatus may detect one or more of a substance, such as a metal,explosive or other chemical substance; a feature identifying the person36, such as a retinal image, facial image, fingerprint, volumetric ortopographical representation of all or a portion of the body, otherbiometric feature, an identification or categorization badge orinsignia, such as an insignia representing that the person belongs to acertain group, such as a security or police force, or radio frequencyidentification (RFID) device. Trace element detection can be based onion mass spectrometry, quadrupole resonance, Fourier-transform infraredspectroscopy, vapor detection, chemical biological sensing, lasermeasurement, fluorescence, DNA analysis, and MEMS. Optionally, asupplemental source 30 may include context data stored in a memory.Context data relates to a person or potential objects carried by theperson and/or subject-associated item(s) associated with the person,such as historical data relating to information previously detected orinput about the particular person, any clearance the person might haveto carry particular types of objects, such as guns or other security orlaw-enforcement devices, or other data that is considered to assistoperators of surveillance system 20 in detecting and assessing thesignificance of objects carried by the surveilled person and/orsubject-associated item(s) associated with the person.

Controller 21 can include a processor 40 in the form of any suitablecomponent for processing the image data, such as digital circuitry,analog circuitry, or both. Processor 40 may be programmable, such as acomputer or microprocessor, a dedicated hardwired state machine, a logicunit, or a combination of these. For a multiple processor form,distributed, pipelined, and/or parallel processing can be utilized asappropriate.

Processor 40 may include a memory 42. Memory 42 can be solid state,electromagnetic, optical, or a combination of these, and it can bevolatile and/or nonvolatile. Further, it may be dedicated, such as aresident hard drive, a random access memory (RAM), or a removable, suchas a removable memory device (R.M.D.). Presently typical removablememory devices include a floppy disc, tape cartridge, optical disc(CD-ROM or DVD), or other appropriate type. Memory 42 also may be aremote memory coupled to the processor by a wired or wirelesscommunication link via another processor and/or network.

Controller 21 may be coupled to suitable input and/or output devices 44adapted to receive data and transmit output data, such as acommunication link to another system or other remote output device.Exemplary input devices may include a keyboard, mouse or othercursor-control device, a voice recognition input system, or otherappropriate device or system. Output devices may include visual or audioalarms, printers, communications links, or displays, such as cathode raytube (CRT), Liquid Crystal Display (LCD), plasma, organic light emittingdiode (OLED), or other appropriate display type. Appropriatecommunication links may include local area networks (LANs), municipalarea networks (MANs), or Wide Area Networks (WANs), whether private orpublic.

A general flow chart 50 illustrating exemplary operation of surveillancesystem 20 is shown in FIG. 2. Two data acquisition phases areillustrated. Where one or more of the subject information sources 24 areimaging apparatus 28, each imaging apparatus detects image informationand produces a detected signal. Imaging apparatus can include apparatusthat detects information relatable to zones or positions of the subject,such as subject surfaces or characteristics, as may be realized, such asusing received radiation of appropriate form, such as acoustical waves,optical radiation, infrared radiation, millimeter-wave radiation orother radio-frequency radiation, Terahertz radiation, and x-rayradiation. The image signals are acquired for each imaging apparatus at52. The acquired image signal is then processed at 54 to construct imagedata. Image features are then identified at 56. As is explained furtherbelow, image features derived from image data may be the shape,configuration, arrangement, or location of one or more objects 38relative to a person 36.

Where one or more subject information sources 24 is a non-imaging source58, the data from the non-imaging source is acquired at 60. Anon-imaging source may be a sensor that detects general features of thesubject, such as the general detection of a substance, a featureidentifying the person 36, or context data stored in a memory relatingto the subject. Image features are then identified at 56. The detectionof the existence of a substance or an identification of the person or, acharacteristic, class or categorization of the person, and otherappropriate indicators or information are considered features of thesubject.

Image features from the various subject information sources 24 are thencorrelated with each other at 62. For example, the identification of anobject on the side of a person from an imaging apparatus may becorrelated with the detection of metal in the middle zone of the person,a badge identifying the person, and context data previously stored inmemory indicating that the person is a security guard and has highsecurity clearance.

These correlated features may then be classified at 64 based on thecombination of features. Classification of features is a logical processfor determining the likelihood that a detected feature is a suspectobject or a false alarm. For example, the detection of metal in the samezone as an object indicates that there is a high likelihood that theobject is metal. Further, given that the person is identified as asecurity guard, it is highly likely that the object is a gun. Also, theperson may be authorized to carry a gun in this position as part of herduties. The object would thus be given a high weight of being a suspectobject, but given a low weight as a security risk, due to the status ofthe person as a security guard.

The processor may use a relational database to relate the variouspossible features relating to a subject that is surveilled. As a generalexample, the following truth table illustrates how two or threefeatures, identified as features A, B, and C, may be related. Based on adetermined relationship, an attribute, weight, or value, Vx, may beassigned to each combination. In this example, reference is made to avalue, which term is intended to include attribute, weight, or otherrelative indicator. These values may be different or the same fordifferent relationships, depending the application.

Does Condition Exist? A B C A {circumflex over ( )} B Value A{circumflex over ( )} (B {circumflex over ( )} C) Value Yes Yes Yes A(B) V₁ A (B {circumflex over ( )} C) V₅ Yes Yes No A (B {circumflex over( )} C) V₆ Yes No Yes A ( B) V₂ A ( B {circumflex over ( )} C) V₇ Yes NoNo Ā ( B {circumflex over ( )} C) V₈ No Yes Yes Ā (B) V₃ Ā (B{circumflex over ( )} C) V₉ No Yes No Ā (B {circumflex over ( )} C) V₁₀No No Yes Ā ( B) V₄ Ā ( B {circumflex over ( )} C) V₁₁ No No No Ā ( B{circumflex over ( )} C) V₁₂ A (B) = A has the condition B Example: IfA: image anomaly B: metal is detected C: explosive is detected Ā: noanomaly detected B: no metal detected C: no explosive detected

Any set of corresponding features can be assigned a correspondingrelative indicator, such as weight, value or attribute. For instance, anobject identified in both a visible image and a millimeter-wave-basedimage can have a low value, since the object is visible. On the otherhand an object identified on a millimeter-wave-based image that is notvisible, can be assigned a high value. Infrared sensors can also be usedto confirm the existence of hidden objects, and can identify a hiddenobject that is not detected on a subject using millimeter-wave-basedimaging. An area of a subject can thus be assigned a high value eventhough no image object is detected. For example, a sheet of plasticexplosive taped to a person's body may appear “colder” than the rest ofthe person's body. Also, the combination of explosive and metaldetectors, may be used to identify an imaged object as likely to be aplastic bomb with metal pieces. Such an object may then be given a veryhigh value.

An object's (or subject area's) value may thus be based on thesubstances the object (or subject) includes, such as metal, plastic,dielectric, ceramic, tissue, fabric, paper, stone, plant matter. Thestructure of the object also can be the basis of assigning a value, suchas dimensions, shape and edge characteristics. The chemical make-up ofan object can also be the basis for assigning a value, including drugs,and explosives, such as PETN, TNT, plastic, C-4 and datasheet.

Once the image features are classified, then conclusions are generatedat 66 about the combinations of image features.

A(B̂C): detected anomaly is metal and is explosive.

A(B̂ C): detected anomaly is metal and is not explosive.

A( B̂C): detected anomaly is not metal and is explosive.

A( B̂ C): detected anomaly is not met and is not explosive.

Ā(B̂C): undetected anomaly is metal and is explosive.

Ā(B̂ C): undetected anomaly is metal and is not explosive.

Ā( B̂C): undetected anomaly is not metal and is explosive.

Ā( B̂ C): undetected anomaly is not metal and is not explosive.

The various conclusions may then be output at 68, as appropriate, suchas via a display, report or alarm condition. However, the stepsdiscussed above may be performed in different sequences and in differentcombinations, not all steps being required for all embodiments ofsurveillance system 20.

There are various ways in which a subject information assembly can beconfigured. A surveillance system, in which sensor apparatus are mountedrelative to a subject position and relative to the different sensorapparatus, can provide for different moving or non-movingconfigurations. For example, a given sensor apparatus may be adapted tomove relative to the subject position but not move relative to one ormore other sensor apparatus. FIG. 3 illustrates various possibleconfigurations for a subject-information assembly 22 that includes threesensor apparatus 26 mounted relative to a subject 32 in a subjectposition 34. Optionally, the sensor apparatus may be mounted for motionrelative to the subject position and to one another by motion apparatus80. A motion apparatus is any appropriate device that providesmechanical motion of a sensor apparatus relative to the subject positionor relative to another sensor apparatus or another motion apparatus.Examples of possible motion apparatus include a motor acting on a pinionguided along a rack, or a stepper motor moving a sensor apparatus alonga track.

As used herein, moving a specified element M relative to anotherspecified element N means the same thing as moving element N relative toelement M. For example, element M could be fixed to a frame, and elementM could be moved relative to element N by moving element N relative tothe frame.

An arrow between two components shown in FIG. 3 represents acorresponding relationship. For instance, an arrow directly between thesubject position 34 and a sensor apparatus 26 is intended to representthat the sensor apparatus does not move relative to the subjectposition. On the other hand, an arrow extending between the subjectposition and a motion apparatus, and between the motion apparatus and asensor apparatus, indicates that the sensor apparatus moves relative tothe subject position. This could be accomplished, for instance, byrotating the subject position on a floor and holding the sensorapparatus fixed relative to the floor, by holding the subject positionfixed relative to the floor and moving the sensor apparatus along thefloor, or by moving independently both the subject position and thesensor apparatus relative to the floor.

There are numerous possible configurations, a portion of which are shownin the figure. For example, where all three sensor apparatus are adaptedto move relative to the subject position, this can be provided by one,two or three (or more) motion apparatus. Where two or three motionapparatus are used, the second and subsequent motion apparatus may bemounted relative to one or more other motion apparatus, therebypotentially providing for the capability of moving concurrently inmultiple directions, depending on how the respective movement apparatusare operated.

FIG. 4 illustrates an optional configuration in block diagram form of asubject information assembly 22 that may be used in a surveillancesystem 20. Subject information assembly 22 may include a base frame 92,which may be a floor or other work surface, relative to which a firstsensor apparatus 26 is directly mounted. A first motor or other motionapparatus 80 is mounted for providing relative motion between the baseframe and a second sensor apparatus 26. A subject platform 94, whichdefines the subject position 34, may then be mounted relative to asecond motion apparatus 80 mounted in turn relative to base frame 92.

An example of a surveillance system 20 having a subject-informationassembly, 22 configured as shown in FIG. 4, is depicted in FIG. 5.System 20 of FIG. 5 includes a controller 21 adapted to control asubject information assembly 22 mounted in a surveillance station orportal 100 defining a subject position 34. In this example, subjectinformation assembly 22 includes a first sensor apparatus 26 in the formof an active, millimeter-wave imaging apparatus 102, adapted tointerrogate subject 32 by illuminating it with electromagnetic radiationin the 100 Megahertz (MHz) to 2 Terahertz (THz) frequency range anddetecting the reflected radiation, as has been described.

As illustrated in FIG. 5, subject 32 includes a person 36 presented forinterrogation by system 20. Person 36 is shown wearing clothing 104,which conceals object 38, shown in the form of a weapon. Subject 32 ispositioned in interrogation station or portal 100 of system 30. Portal100 may be configured for placement at a security checkpoint where it isdesired to detect objects, such as weapons or contraband, on the person.Portal 100 may include a platform 106 connected to a motion apparatus 80in the form of a motor 108. Platform 106 may be arranged to supportsubject 32. Motor 108 may be arranged to selectively rotate aboutrotational axis R while subject 32 is positioned thereon. For theconfiguration shown, axis R may be vertical, and subject 32 may be in agenerally central subject position 34 relative to axis R and platform106.

Imaging apparatus 102 may include an antenna apparatus 110 including aprimary multiple-element sensing array 112. Subject information assembly22 may include a frame 114 on which array 112 may be supported. Array112 may extend the full height of frame 114. Motor 108 causes platform106, and subject 32 to rotate about axis R. As a result, array 112circumscribes a generally circular pathway about axis R. The antennaarray may be about 0.5 to about 2 meters from radius R.

Antenna array 112 may include a number of linearly arranged elements 116only a few of which are schematically illustrated. Each element 116 maybe dedicated to transmission or reception of radiation, and the elementsmay be arranged in two generally vertical columns, with one columndedicated to transmission, and the other to reception. The number andspacing of the elements corresponds to the wavelengths used and theresolution desired. A range of 200 to about 600 elements can span avertical length of about two or two and one-half meters.

Subject information assembly 22 includes a second sensor apparatus 118.Optionally, additional sensor apparatus may be mounted to second sensorapparatus 118, such as a third sensor apparatus 120. The second sensorapparatus may be mounted by a prime mover 122, for vertical movementalong frame 114. Thus, as platform 106 rotates about axis R, sensorapparatus 118 scans the zone or location of the subject correspondingwith the vertical position of the sensor and the rotational position ofthe subject.

Various other configurations for portal 100 and antenna apparatus 110may be used. For example, a two-dimensional transmit and receive arraymay be used, as well as an array that moves around a fixed subjectplatform, or an array that moves vertically and extends horizontally.Further, many variations of an antenna apparatus are possible. Theantenna apparatus may include one or more antenna units, and eachantenna unit may include one or more transmitting antennae and one ormore receiving antennae. An antenna unit may include a plurality ofantennae that may receive radiation in response to transmission by asingle antenna. The antennae may be any appropriate type configured totransmit or receive electromagnetic radiation, such as a slot line,patch, endfire, waveguide, dipole, semiconductor, or laser. Antennae mayboth transmit and receive. The antennae units may have one or moreindividual antennae that transmit or receive like polarization or unlikepolarized waveforms such as plane, elliptical, or circular polarization,and may have narrow or broad angular radiation beam patterns, dependingon the application. Beam width may be relatively broad, i.e. 30 to 120degrees for imaging applications that use holographic techniques, whilenarrow beam widths in the range of 0 to 30 degrees may be used forapplications having a narrow field of view requirement.

Further, a single antenna may scan a subject by mechanically movingabout the subject in a one- or two-dimensional path. A one- ortwo-dimensional array of antenna units may electronically andmechanically scan a subject. An imaging system may include one or aplurality of antenna apparatus. The antennae apparatus may be protectedfrom the environment by suitable radome material, which may be part ofthe apparatus, or separate, depending on the mechanical motion that isrequired of the antennae apparatus or array. Examples of other arrayconfigurations are illustrated in copending patent application Ser. No.10/728,456 filed Dec. 5, 2003, entitled “Millimeter-Wave Active ImagingSystem” incorporated herein by reference.

Antenna apparatus 48 may be configured to transmit and receiveelectromagnetic radiation selected from the range of about 1 Gigahertz(GHz) to about 2 THz, or from the range of about 100 MHz to about 15GHz, depending on the impulse duration. A range of about 1 GHz to about300 GHz may also be used, with a range of about 5 GHz to about 110 GHzparticularly useful for imaging. A range of 24 GHz to 30 GHz is used insome antenna apparatus. The antenna apparatus produces an image signal68 representative of the received radiation.

Various configurations of second and third sensor apparatus are alsopossible. For example, a plurality of sensor apparatus 118 may bemounted along antenna array 112 in fixed positions. Each sensorapparatus 118 or 120 may have a sensor unit 121 (such as a receiver) onone side of subject position 34 and a complementary unit (such as atransmitter) on an opposite side or other spaced position, asrepresented by sensor unit 123.

A controller 21 may control operation of subject information assembly22. Controller 21 may include a transceiver 124 including a switchingtree 126 configured to irradiate subject 32 with only one transmittingelement 116 at a time, and simultaneously receive with one or moreelements 116. Transceiver 124 may include logic to direct successiveactivation of each combination of transmit and receive antenna elementsto provide a scan of a portion of a subject 32 along a verticaldirection as platform 106 and the subject rotate.

An image signal 125 received from antenna apparatus 110 may bedownshifted in frequency and converted into an appropriate format forprocessing. In one form, transceiver 124 may be of a bi-staticheterodyne Frequency Modulated Continuous Wave (FM/CW) type like thatdescribed in U.S. Pat. No. 5,859,609. Other examples are described inU.S. Pat. Nos. 5,557,283 and 5,455,590. In other embodiments, a mixtureof different transceiver and sensing element configurations withoverlapping or non-overlapping frequency ranges may be utilized, and mayinclude one or more of the impulse type, monostable homodyne type,bi-static heterodyne type, and/or other appropriate type.

Transceiver 124 may provide image data 127 corresponding to the imagesignals to one or more processors 40. Processor 40 can include anysuitable component for processing the image data, as appropriate.Processor 40 is coupled to a memory 42 of an appropriate type andnumber. As has been mentioned, memory 42 may include a removable memorydevice 129, as well as other types of memory devices.

Controller 21 may be coupled to motor 108, prime mover 122, or otherdrive element used, to selectively control the rotation of platform 106or movement of a sensor apparatus 26. Controller 21 may be housed in amonitor and control station 128 that also includes one or moreinput/output devices 44, such as operator input devices 130 and one ormore display or other type of output devices 132.

FIG. 6 illustrates a simplistic side view of another embodiment of asubject information assembly 22, usable in surveillance system 20 as hasbeen described. In this example, subject information assembly 22 has abase frame 140 including left upright 142 and right upright 144. A firstinformation source 24 is a sensor apparatus 26 in the form of an imagingapparatus 28, and in particular, an antenna apparatus 110, such as wasdescribed with reference to FIG. 5.

Antenna apparatus 110 includes array elements 116 that may be mounted bya primary motion apparatus, not shown, to frame 140 for movingvertically. A second sensor apparatus 26 may be mounted relative to anintermediate frame 146 by a second motion apparatus, also not shown. Inturn, intermediate frame 146 may be mounted relative to base frame 140by a further motion assembly, also not shown. The subject informationassembly 22 shown in FIG. 6 thus may provide for independent movement ofthe first and second sensor apparatus 26. Optionally, a second sensorapparatus 26′ may be mounted on an intermediate frame 146′ for movementwith the first sensor apparatus 26 and relative to (along) the antennaapparatus 110.

A simplified top view of a further variation of a subject informationassembly 22 for a portal 100 and usable in a surveillance system 20, isshown in FIG. 7. In this example, subject information sources 24 includefirst and second sensor apparatus 26. A first sensor apparatus 26′includes an active millimeter-wave imaging apparatus 28, as describedabove, having an antenna apparatus 110′ formed as a horizontal array112′. Array 112′ may have various configurations, and is shown witharray elements 116 extending in a convenient arc 150 around a subject 32in a subject position 34.

Array 112′ is shown mounted on a frame 114 having opposite distal ends114 a and 114 b. Second sensor apparatus 26″ also includes a firstsensor unit 152 and a second sensor unit 154 mounted, respectively, onframe ends 114 a and 114 b. Sensor units 152 and 254 may be the sametype of sensor, or different types of sensors, when a single sensor unitis sufficient. If sensor apparatus 26″ is of a type that requires a sendor transmit unit and a spaced-apart complementary receive unit, then thetwo sensor units can be the two components required. Examples of suchsensor apparatus include metal detectors, some imaging systems, andtrace chemical detectors. With trace chemical detectors, one sensor unitmay direct air toward the subject, such as by a fan, and a second sensorunit draws in the air, which air is then checked for the trace elementor elements of interest.

By traveling vertically with the imaging array 112, the second sensorapparatus 26″ senses subject information at a level corresponding withthe position of the imaging array. As a result, characteristics of theimage in the position or zone of the image corresponding to whichsupplemental subject information are detected, can be correlated withthe subject information to provide further information relating to thedetected image of the subject. Other configurations for the secondsensor apparatus can also be realized. For example, the sensor units maybe mounted in fixed positions on a fixed portion of frame 114. Also, aplurality of sensor units can be distributed vertically along fixedportions of frame 114, and mounted in fixed positions independent ofarray 112′, to provide zonal subject information. Optionally, the sensorunits may be mounted independent of imaging apparatus 28′, and may moveor not, as was discussed with reference to subject information assembly22 shown in FIG. 3. For instance, a sensor unit may be a device forreceiving a card, token or other item handled by the surveilled person,which item is then tested for a trace element. Another technique forobtaining a trace element would be to mount the sensor units in handlesthat the person grasps during imaging. The surface of the handles couldbe wiped or removed and tested for trace elements, or air could be drawnin through or around the handle for testing.

A motion apparatus 80 is adapted to move the first and second sensorapparatus 26 vertically. Accordingly, a subject 32 may be imaged on oneside by moving the array vertically from top to bottom or from bottom totop. In order to image both sides of a subject, then, it is necessary toscan the opposite side of the subject. This can be accomplished invarious ways, such as having the subject turn around. Optionally, asecond array, opposite the array shown, could be provided. Also, thearray shown could be rotated around the subject position.

Following is a description that illustrates a method for providing ablurred image of a subject for protection of privacy concerns ofsurveilled subjects, as well as for automatic identification of portionsof an image suspected of including objects carried by the surveilledperson. This description is presented largely in terms of displayimages, algorithms, and symbolic representations of operation of databits within a computer memory. It may be preferred to implement aprogram or algorithm as various interconnected distinct software modulesor features. This is not necessary, as software, firmware, and hardwaremay be configured many different ways, and may be aggregated into asingle processor and program with unclear boundaries.

An algorithm is generally considered to be a self-consistent sequence ofsteps leading to a desired result. These steps require physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, transferred, combined, compared, and otherwisemanipulated. When stored, they can be stored, transferred, combined,compared, and otherwise manipulated. When stored, they may be stored inany computer-readable medium. As a convention, these signals may bereferred to as data, bits, values, elements, symbols, characters,images, terms, numbers, or the like. These and similar terms may beassociated with appropriate physical quantities and are convenientlabels applied to these quantities.

The present disclosure also relates to apparatus for performing theseoperations, such as has been described. This apparatus may be speciallyconstructed for the required purposes or it may comprise ageneral-purpose computer selectively activated or reconfigured by acomputer program stored in the computer or other apparatus. Inparticular, various general-purpose machines may be used with programsin accordance with the teachings herein, or it may prove more convenientto construct more specialized apparatus to perform the required methodsteps. The required structure for a variety of these machines willappear from the description given below. Useful machines for performingthe disclosed operations include general-purpose digital computers,microprocessors, or other similar devices, as has been described.

The programs described need not reside in a single memory, or even asingle machine. Various portions, modules or features of it can residein separate memories, or even separate machines. The separate machinesmay be connected directly, or through a network, such as a local accessnetwork (LAN), or a global or wide area network, such as what is knownas the Internet. Similarly, the users need not be collocated with eachother, but each only with a machine that houses a portion of theprogram.

Various techniques may be used for identifying objects that may beassociated with a person being imaged. For example, once a first imageof a subject has been obtained, objects are detected visually by anoperator of the system or automatically. When manual detection is used,a primary or first image 160, such as the image of FIG. 8, may bedisplayed on a system monitor. An operator may then identify positionsor areas of the image that are suspected of being objects. For example,FIG. 9 illustrates an enlarged second image 162 of a portion of theimage of FIG. 8 as defined by an outline 164 around the suspect area.Such an area may be defined and the enlarged image produced usingimage-enhancing algorithms. Such a program may also provide for imageenhancement and increased resolution, in order to produce a clearerimage of the region in a selected suspect area. For instance, theprogram may increase the number of picture elements or pixels used toimage the suspect area. Image enhancement tools, such as interpolationtechniques, may be used to make the image sharper.

Once a suspect area containing a possible object other than the person,is identified, the antenna apparatus may be used to provide a moredetailed image of the suspect area. For instance, a secondary antennaarray, not shown, may be used to re-image the suspect area in greaterdetail than was originally provided.

Optionally, imaging arrays may be used to re-image the suspect area.Increased resolution may be possible by increasing the firings persurface area of the subject, such as by reducing the rate of movement ofthe subject relative to the antenna array, or increasing the scanningrate for the antenna array. It may also be possible to produce thefirst, low-resolution image using a portion of antenna elements in thearray, and to produce a second, higher-resolution image using all of theantenna elements corresponding to the selected region of the subject.

Accordingly, the image signal, whether it be a reworking of the portionof the image signal corresponding to the first image or a portion of theimage signal corresponding to a rescan of the selected subject area, asecondary image may be produced that has a higher resolution than theprimary image. This image may then be displayed for observation by anoperator, to better determine whether an object is present, and if so,the nature of the object.

Characteristics of image data may also be automatically analyzed todetermine whether there may be areas of an image that may include anobject. Regions of a subject image that have characteristics thatcorrespond to characteristics of images of known objects may beidentified. By automatically identifying suspect areas of an image, asystem operator's attention may be directed to such areas, theinformation may be used to activate alarms, and the information may beused to initiate a rescanning of the suspect areas or to otherwiseincrease the resolution of the image.

It has been found that objects carried by a person may have theappearance of a “wavy texture” in the vicinity of objects hidden on aperson. FIG. 8 shows an example of an image 160 of a mannequin withman-made objects placed on its body. This image is comparable to animage of a person. This image is comparable to an image of a person. Inthe figure, the image of a knife appears on the abdomen, and an image ofa gun appears on the back. Second image 162 shown in FIG. 9 is anenlargement of the image of the knife shown in FIG. 8. An imagecharacterized as having picture elements with alternating high and lowintensity levels, appearing as light and dark regions, may be suspectedof being an object. This varying intensity level pattern or wavy texturepattern is visible in the image on and/or around the object.

The waviness may be detected by convolving the image with a wavinessdetection kernel. An exemplary kernel is given by the followingfive-by-five matrix of values:

$\quad\begin{matrix}0.0278 & {- 0.1111} & 0.1667 & {- 0.1111} & 0.0278 \\{- 0.1111} & 0.4444 & {- 0.6667} & 0.4444 & {- 0.1111} \\0.1667 & {- 0.6667} & 1.0000 & {- 0.6667} & 0.1667 \\{- 0.1111} & 0.4444 & {- 0.6667} & 0.4444 & {- 0.1111} \\0.0278 & {- 0.1111} & 0.1667 & {- 0.1111} & 0.0278\end{matrix}$

The “wavy” nature of the above set of numbers is illustrated graphicallyin FIG. 10, which illustrates the relative magnitudes of afive-row-by-five-column grid corresponding to pixel positions. Thevalues in the rows and columns of the kernel shift between higher andlower values, or in this case, between negative and positive values,hence the term “wavy”.

The kernel numbers above can be thought of as a small image windowcovering a corresponding five-by-five square of picture elements, whichwindow is moved above (associated with successive corresponding portionsof) the image. Each value in the kernel is multiplied by thecorresponding image intensity value (right below it) and the finalresult of the operation is the sum of the results of themultiplications. Generally, the closer the relative values of theintensities of the pixels overlaid by the window are to the relativevalues in the kernel, the higher the magnitude or absolute value of theresulting sum.

An illustration of this operation is depicted in FIGS. 11A and 11B,which show two consecutive positions of the kernel window. In thesefigures, the circles 170 represent the intensity levels of pixels makingup a subject image 172. The square 174 represents the “window” of thekernel, with the kernel values represented by the triangles 176. Thetriangles 176′ shown in dashed lines to the left of square 174 simplyrepresent a row of pixels that were included in window 174 during theprevious computation. As mentioned, each kernel value is multiplied bythe intensity level of the pixel it is associated with in a givenposition of the window. The resulting 25 products are summed. Theabsolute value of the sum is assigned to a reference pixel R, which maybe any pixel position considered appropriate. In this example, the pixelin the center of the window is considered the reference pixel.

This sum provides an indication of the correlation of the 25 pixelsassociated with the reference pixel to the relative variations ortexture represented by the kernel. The higher the absolute value of thesum, then the higher the correlation.

FIG. 11B shows the position of window 174 during the next step in whichthe window is shifted one pixel row to the right. This process continuesuntil the subject image 172 is convolved into a new “image” with itsvalues containing the results of the above operation for each pixel orselected pixels of the image. After smoothing and stretching to occupy afixed dynamic range, the new image is a “waviness image” 180 shown inFIG. 12, in which the lighter areas indicate the local presence of thewaviness texture. In other words, the image of FIG. 12 is a mapping ofthe extent the selected subject-image characteristic is present. It isseen in the waviness image that the biggest and brightest regions ofwaviness appear where the knife and gun are in the original subjectimage.

Once the waviness image is produced from the original subject image, thebrightest regions of the waviness image may be identified by running athreshold analysis on the image. Small contrasting regions or holes mayalso be eliminated in order to make the resulting image more uniform. Inthis analysis, a new image 182 as shown in FIG. 13 is produced byassigning a distinct intensity level, such as that corresponding towhite, to those regions of the waviness image having at least a selectedintensity or waviness value, such as 80 percent of maximum. The otherregions are assigned a distinguishable value, such as zero or blacklevel intensity. This image then shows those regions in which strongcorrelation exists for the selected image characteristic.

The intent is to identify graphically those regions of the subject imagewhere there are suspected objects. These regions can then be displayedor otherwise identified to a system operator, so that they can evaluatethe images or the subject further. The above process may result in minorregions being identified for which it is unlikely that they correspondwith an object on the person. The image may accordingly be cleaned up bydilating the highlighted spots and then deleting smaller regions notexpected to be significant. Dilating is a process of expandinghighlighted regions. One way of doing this is by passing another“window” of analysis or kernel over the image. For a given referencepixel, if there are more white pixels than a preset threshold, then thepixels in the entire window are made white. An exemplary result is shownas dilated image 184 in FIG. 14.

To simplify the imaging of the suspected object areas, rectangles thatbound the highlighted regions may be identified. FIG. 15 illustrates animage 186 that results from such a process. The rectangular regions arelarger in size than the previously highlighted regions. Also, someregions may include more than one suspected area.

This image may again be dilated to further combine closely spaced areas.The image 188 that results from one such process is shown in FIG. 16. Itis seen that in this case, the image of FIG. 16 is nearly the same asthe image of FIG. 15. Different criteria for dilating may producedifferent results. This image may then be used to produce a final maskof the suspect areas.

To facilitate the identification of objects on the subject image, theoutlines of the suspect areas, derived from the image of FIG. 16, may beadded to a display of the original image, as shown in FIG. 17. Image 190shown in this figure is a reproduction of original image 160 shown inFIG. 8, with outlines of the suspect regions as defined by image 188.Optionally, as is shown in FIGS. 18 and 19 below, the portions of theoriginal subject image within the suspect areas may be superimposed on amodified subject image, such as image 180 of FIG. 12, to preserveprivacy concerns of a subject person.

As has been mentioned, a surveillance system 20 can be configured toprovide both an image of a subject, as well as supplemental subjectinformation. The source of supplemental subject information can be inthe form of a sensor, such as a metal detector. As described withreference to the subject-information assembly 22 illustrated in FIGS.5-7, metal can be detected in regions of the subject. These regions maybe a plurality of vertically distributed zones, as may be provided bysensor apparatus 26″ depicted in FIG. 7.

FIG. 18 illustrates a display 200 of an output device 132 of asurveillance system 20. Display 200 may be a computer-generated displayof a control panel 202, displayed subject images 204, and a supplementalsensor output display 206. Control panel 202 may include a variety ofdisplay and system operation controls 208, such as control keys 210.

Various configurations of a subject image or images 204 can be provided.In this example, eight frames 212 of images corresponding to views ofthe subject at 45-degree increments around the subject are shown. Eachframe includes a subject image 214, which in this instance is a privacyimage 216 of the subject, such as provided by the process described withreference to FIGS. 8-17. Suspect regions 218 of the detailed subjectimage, such as regions identified by an object-identification algorithm,as discussed above, are super-imposed on the privacy images. A selectedone of the eight frames is enlarged and displayed as a primary image220. The enlarged frame includes enlargements of the suspect regions,making them easier to see by an operator.

Supplemental display 206 visually associates the results of operation ofa second sensor apparatus 26. In this example, the second sensorapparatus may be a metal detector that determines the existence of metalin different vertical zones of the person's body. A display element 222is aligned with each of the detected zones of the image 220. The displayelement in this case has a positive state, represented by a darkindication or distinctive color, when metal is detected in thecorresponding zone of the subject. When no metal is detected for aparticular zone, the display element has a negative state represented bya light indication or distinctive color. Other symbols, indicia orrepresentations could be used to indicate the results of the secondsensor apparatus relative to the subject image.

It is seen that for the selected primary image 220 shown in FIG. 18,suspect regions 218 exist in the pelvic, lower torso and head regions.The top and third down metal display elements 222 are shown to be in thepositive state and the others are in the negative state, indicating thatone or more of the objects identified in each of these two zones is orare made of metal. Based on this result, the surveilled person could bewatched or apprehended, to prevent entry into an associated facility, ortake any other action deemed appropriate in the circumstances.

When there is a match or correlation of detected information in aparticular location or particular locations of a subject, one or moreimages may be further enlarged or cropped to focus attention on andprovide additional detail of the suspect regions or zones where metal isdetected. An additional display 230 that may be provided, is shown inFIG. 19. This display generally has the same features as display 200,and corresponding features are shown with a prime. The image frames 212′and primary image 220′ are rearranged, enlarged, and/or cropped to showsuspect regions 218, and eliminate continuous zones that do not have asuspect region or an indication of positive metal detection. Theenlarged images, such as enlarged primary image 232, may be providedwithout further alteration, or may provide increased resolution. Theimage resolution can be increased by image-enhancement or editingsoftware, as well as by rescanning the subject with the same array in ahigher resolution mode, or with a finer-resolution array.

Although not shown in the figures, a surveillance system may also beadapted to provide images of regions of a subject for which no suspectregions are identified, but for which a supplemental sensor detects asubstance or other characteristic. In the case where metal is detectedin a zone for which no suspect regions are identified, an enlargement ofthe image of the affected zone provides improved resolution or detail,to show whether an external object is present in that zone.

Another example of a general surveillance system is shown in FIG. 20 andis generally indicated at 300. Surveillance system 300 may include atleast one controller 302 and a subject-related information assembly 304.“Subject-related information,” as used herein, refers to informationrelated to a subject and/or one or more subject-associated items. Unlessotherwise indicated, surveillance system 300 may include one or more ofthe structures and/or functions of structures of general surveillancesystem 20 discussed above.

Subject-related information assembly 304 may include a subjectinformation assembly 306 and a subject-associated item informationassembly 308. The subject information assembly may include one or moreof the structures and/or functions of structures of subject informationassembly 22 discussed above. For example, subject information assembly306 may include a plurality of subject information sources 310, whichmay at least partially include the structures and/or functions of theone or a plurality of screening apparatus 26 and/or the one or aplurality of supplemental subject information sources 30. The subjectinformation sources may provide information related to subject 32 insubject position 34, such as information on person 36 and objects 38with and/or supported on the person. Subject information sources 310 maybe controllable or controlled by controller 302.

Subject-associated item information assembly 308 may include a pluralityof subject-associated item information sources 312. Thesubject-associated item information sources may provide informationrelated to one or more subject-associated items 314, such as any objects316 carried by those items. “Subject-associated items,” as used herein,refers to items associated with the subject but not part of the subject,such as divested objects, carryon baggage, and checked baggage.Subject-associated item information sources 312 may be controllable orcontrolled by controller 302.

The subject-associated item information sources may include at least onesubject-associated item screening apparatus 318. Screening apparatus 318may be configured to screen at least a portion or a plurality ofsections of the one or more subject-associated items and/or producescreening data or a screening signal representative of screening the atleast a portion of the one or more subject-associated items. In someembodiments, screening apparatus 318 may be adapted to detect a givencharacteristic (and/or object information) of an object potentiallycarried by one or more subject-associated items and to produce ascreening signal representative of the detection of that characteristic(and/or screening data indicating the detection of the characteristic).In some embodiments, screening apparatus 318 may be adapted to produceobject data indicative of whether the subject-associated item(s) carriesat least one object that includes the object information or a screeningsignal representative of the detection of the given characteristic. Thescreening data may be received by controller 302 and/or other structuresof surveillance system 300, as further discussed below.

In some embodiments, the subject-associated item screening apparatus mayinclude at least one subject-associated item imaging apparatus 320,which may include any suitable structure adapted to image at least aportion of the subject-associated item(s) and/or to produce an imagesignal representative of an image of the at least a portion of thesubject-associated item(s). For example, imaging apparatus 320 may beadapted to transmit, toward the one or more subject-associated items,X-ray electromagnetic radiation for use in X-ray transmission,diffraction, and/or backscattering imaging. In some examples, the X-rayradiation may be in a frequency range of about 30 Petahertz (PHz) toabout 30,000 PHz, with corresponding wavelengths that may range from 10to 0.01 nanometers. Examples of the use of X-ray electromagneticradiation for screening subject-associated item(s) are further describedin U.S. Pat. No. 5,600,303 to Husseiny et al., the complete disclosureof which is herein incorporated by reference for all purposes.

Imaging apparatus 320 may alternatively, or additionally, include othersuitable imaging technologies such as computed tomography (CT), magneticresonance imaging (MRI), smart X-ray, multi-view X-ray, laminography,and/or other technologies discussed above. Structures and features ofsome examples of imaging apparatus 320 are further described in U.S.Pat. No. 6,952,163 to Huey et al., the complete disclosure of which isherein incorporated by reference for all purposes. Although imagingapparatus 320 is discussed to incorporate or use particulartechnologies, the imaging apparatus may alternatively, or additionally,include one or more suitable technologies configured to image thecontents of the subject-associated item(s).

Screening apparatus 318 may alternatively, or additionally, be adaptedto detect, in the one or more subject-associated items, one or more of ametal substance, an explosive substance (such as Semtex, C-4,nitroglycerin, PETN, RDX, Detasheet, TNT, tetryl, ANFO, and blackpowder), a chemical substance (such as illegal drugs, including cocaine,heroin, and MDMA), and a feature identifying the person associated withthe one or more subject-associated items (such as one or more tagsattached to the item(s) and having a barcode, RFID, and/or image of theperson). Screening apparatus 318 may include one or more suitabletechnologies. For example, in some embodiments where thesubject-associated item screening apparatus is adapted to detect metal,explosive, and/or chemical substance(s), the screening apparatus mayinclude X-ray transmission, diffraction, and/or backscatter, nuclearquadropole resonance (NQR), trace detection, and/or other suitabletechnologies. The use of X-ray diffraction, NQR, and trace detection inscreening one or more subject-associated items are further discussed inU.S. Pat. No. 5,600,303 to Husseiny et al. and U.S. Pat. No. 6,952,163to Huey et al. Additionally, or alternatively, in some embodiments wherethe subject-associated item screening apparatus is adapted to detect afeature identifying the person associated with the one or moresubject-associated items, the screening apparatus may include barcodereader(s), RFID reader(s), video camera(s), and/or other suitabletechnologies.

Subject-associated item information sources 312 may additionally, oralternatively, include context data stored in a memory. Context datarelates to potential objects carried by the one or moresubject-associated items, such as historical data relating toinformation previously detected or input about the one or moresubject-associated items and/or the person associated with those items,any clearance the person associated with the one or moresubject-associated items might have to carry particular types ofobjects, and/or other data that may be considered to assist in detectingand assessing the significance of objects in the one or moresubject-associated items. The context data may include results ofprevious screening(s) of a particular person and/or theirsubject-associated items within the particular facility or area in whichthe surveillance system is used in (which may be stored, for example, inmemory associated with surveillance system 300) and/or from otherfacilities and areas that have screened the particular person and/ortheir subject-associated item(s). Additionally, or alternatively,context data may include information from various public sources, suchas the FBI, National Crime Information Center (NCIC), State Departmentdatabases, IRS, Social Security Administration, state motor vehicle andcorrections departments, credit bureaus, and/or bank records, and/orprivate sources, such as organizations that generate preferred passengerlists.

Although subject-associated item information sources 312 are shown toinclude screening apparatus 318 and context data, the informationsources may additionally, or alternatively, include other types ofinformation sources configured to provide information regardingpotential objects 316 carried by the one or more subject-associateditems.

Controller 302 may be adapted to operate or control operation of one ormore subject information sources 310 and/or one or moresubject-associated item information sources. For example, when one ormore of those sources include an imaging apparatus and a screeningapparatus, the controller may be adapted to produce, from an imagesignal of the imaging apparatus, image data representative of the imageof at least a portion of a subject and/or subject-associated item(s)(and/or image data representative of detection of a characteristic of anobject potentially carried by a subject and/or one or moresubject-associated items). Additionally, or alternatively, thecontroller may be adapted to produce, from a screening signal of ascreening apparatus, screening data representative of screening at leasta portion of a subject and/or subject-associated item(s) (and/orscreening data indicating detection of a characteristic of an objectpotentially carried by the subject and/or one or more subject associateditems). Alternatively, as discussed above, the imaging apparatus may beconfigured to produce the image data and/or the screening apparatus maybe configured to produce the screening data. Although controller 302 isdescribed as being adapted to produce image data and screening data froman image signal and a screening signal, respectively, the controller maybe adapted to produce data from any suitable signal or combination ofsignals.

Additionally, or alternatively, the controller may be adapted to relatea first screening data to a second screening data, and to produce, fromthe related first and second screening data, relational information datarelating the first screening data and the second screening data. Forexample, the controller may be adapted to relate image data to otherscreening data, and to produce, from the related image data andscreening data, relational information data relating the image data andthe screening data. Although controller 302 is described as beingadapted to relate image data to screening data, the controller may beadapted to relate any suitable combination of data and to producerelational information data relating the combination of data. Forexample, the controller may alternatively, or additionally, be adaptedto relate context data to screening data, and to produce, from therelated context data and screening data, relational information datarelating the context data and the screening data.

Alternatively, or additionally, controller 302 may be adapted to receivedata from one or more subject information sources and/or one or moresubject-associated item information sources, and control one or moreoperating parameters of one or more other subject information sourcesand/or one or more other subject-associated information sources based,at least in part, on the received data. For example, the controller maybe adapted to receive first screening data from a first screeningapparatus and control a sensitivity level of one or more screeningapparatus other than the first screening apparatus. Additionally, oralternatively, the controller may be adapted to select for screening oneor more portions of the subject and/or the subject-associated item(s)(or one or more portions of at least one section of the subject and/orone or more subject-associated items) based, at least in part, on thefirst screening data.

For example, the controller may receive data from screening asubject-associated item indicating that negligible amounts of anexplosive substance were contained in the subject-associated item. Thecontroller may then increase sensitivity levels for the same explosivesubstance when screening the subject and/or other subject-associateditems. Although controller 302 is discussed to be adapted to controlsensitivity level(s) or portion(s) to be screened, the controller may beadapted to control any suitable operating parameter(s) of downstreaminformation sources based, at least in part, on received data from oneor more upstream information sources.

Alternatively, or additionally, controller 302 may be adapted to analyzescreening data, receive information data related to the subject and/orone or more subject-associated items, and then reanalyze the screeningdata based, at least in part, on the information data, and in mannerthat is at least partially different from the first analysis of thescreening data. The controller may, in analyzing the screening data,assign one or more of a relative weight, value, attribute, and relativeindicator to an object potentially with a screened subject and/or inscreened subject-associated item(s) based, at least in part, on thescreening data. Controller 302 may, after obtaining the informationdata, reassign one or more of a relative weight, value, attribute, andrelative indicator to the object potentially with the screened subjectand/or in the screened subject-associated item(s) based, at least inpart, on the obtained information data.

For example, the controller may analyze screening data from screening ofa subject-associated item and conclude that a flammable substance foundin the subject-associated item is a low threat. Controller 302 may thenobtain information data regarding the person associated with thesubject-associated item that identifies the person as a suspectedterrorist. The controller may then reanalyze the screening data with,for example, a higher sensitivity level and conclude that the flammablesubstance is a higher threat based, at least in part, on the obtainedinformation data.

Alternatively, or additionally, the controller may analyze firstscreening data from screening of a first subject-associated item andconclude that a first portion of a potentially dangerous device found inthe subject-associated item is a low threat without a second portion.Controller 302 may then obtain second screening data from screening thesubject and/or other subject-associated item(s) indicating the presenceof the second portion of the device. The controller may then reanalyzethe first screening data and conclude that the first portion of thedevice is a higher threat based, at least in part, on the presence ofthe second portion of the device.

Controller 302 may include at least some of the structures and/orfunctions of controller 21 discussed above. For example, controller 302may include a processor 322, a memory 324, and input and/or outputdevices 326, which may include at least some of the structures describedabove for processor 40, memory 42, and input and/or output devices 44,respectively.

Subject information sources 310 and subject-associated item informationsources 312 may be configured, distributed, and/or arranged in anysuitable manner to obtain information on a subject and/orsubject-associated items. For example, FIG. 21 shows an example of aprocess 400 of boarding a vehicle or conveyance at a vehicle facility(such as an airport, bus terminal, or train station) and howsurveillance system 300 may be used to obtain information regardingsubject 32 and subject-associated items 314 during that process.Subject-associated items 314 may include one or more checked baggage328, one or more divested objects 329, and one or more carryon baggage330. Process 400 involves boarding of subject 32 and subject-associateditems 314 onto a vehicle 332 having one or more controlled-access areas333, such as one or more cargo portions 334 and one or more passengerportions 336. When subject 32 is to be conveyed in a conveyance thatincludes a controlled-access area and in which the subject is traveling,the subject may be referred to as a “passenger.” A controlled-accessarea may also be an area fixed in place, such as represented by abuilding or other facility.

The subject and the subject-associated items begin the process at 402.Checked baggage 328 may be physically separated from the subject at 404,may be screened by at least one checked-baggage information source 338having a processor 340 at 406, and then may be moved to the cargoportion of the vehicle. The subject-associated items that are selectedby the person to be moved to the cargo portion of the vehicle also maybe referred to as “cargo-area items.”

Subject 32 may be physically separated from the subject-associated itemsat 410 and then one or more objects 38 with or supported on person 36may be physically separated from the subject at 412. Those objects maybe referred to as “divested objects” when physically separate from thesubject. The subject may be screened at 414 by at least one subjectinformation source 342 having a processor 344. The divested objects maybe screened by at least one divested object information source 346having a processor 348 at 416. The screened divested objects may bereturned to the screened subject or picked up by the screened subject at418.

Carryon baggage 330 may be physically separated from the subject at 420and may be screened by at least one carryon-baggage information source350 having a processor 352 at 422. The screened carryon baggage may bereturned to the screened subject at 424. The screened subject and thescreened carryon baggage may then move to the passenger portion of thevehicle at 426. The subject-associated items and the divested objectsthat are selected by the person to be carried by the person into thepassenger portion of the vehicle also may be referred to as“passenger-area items” or “carried items.”

One or more of checked-baggage information source 338, subjectinformation source 342, divested object information source 346, andcarryon baggage information source 350 may be at least partiallycontrolled by controller 302 having processor 322 and memory 324.Although those information sources are shown to each have its ownprocessor, one or more of the information sources may not have aseparate processor, but rather may share a processor, such as processor322.

Additionally, or alternatively, at least one of the information sourcesmay be used to screen two or more of the checked baggage, the subject,the divested objects, and the carryon baggage. For example, the at leastone carryon baggage information source may be used to screen the carryonbaggage and the divested objects. Alternatively, or additionally, the atleast one subject information source may be used to screen the subject,the divested objects, and the carryon baggage.

Moreover, although surveillance system 300 is shown to have a particularconfiguration for obtaining information regarding a subject and/orsubject-associated items that are boarding a vehicle with at least onecargo portion 334 and at least one passenger portion 336, thesurveillance system may be configured in any suitable way as appropriatefor the facility, the vehicle, threat level associated with the facilityand/or the vehicle, and/or other suitable factors. An example ofconfiguring the surveillance system would be to exclude one or moreinformation sources shown in FIG. 21 or add one or more other suitableinformation sources. For example, the checked-baggage information sourcemay be excluded if the vehicle includes only a passenger portion 336.Alternatively, the subject, divested object, and carryon-baggageinformation sources may be excluded if the vehicle includes only a cargoportion 334.

Furthermore, although surveillance system 300 is shown to be used in avehicle facility, the general surveillance system may be configured foruse in any type of facility. For example, surveillance system 300 may beused for an entrance of any type of building or facility having one ormore controlled-access areas 333. When the general surveillance systemis used in those applications, the system may exclude the at least onechecked-baggage information source and/or other suitable informationsources.

A flow chart 450 illustrating an example of a method of operation ofsurveillance system 300, is shown in FIG. 22. As discussed above,subject-related information assembly 304 may include one or more subjectinformation sources 310 and one or more subject-associated iteminformation sources 312. Data may be acquired from the subjectinformation sources at 452, and then processed at 454. Similarly, datamay be acquired from the subject-associated information sources at 456,and then processed at 458.

One or more of the subject information sources and/or thesubject-associated item information sources may include an imagingapparatus that produces an image or image data representative of animage of at least a portion of the subject and/or the subject-associateditem(s). The imaging apparatus may include apparatus that detectsinformation relatable to zones, sections, portions, and/or positions ofthe subject and/or the subject-associated item(s), such as surfaces orcharacteristics, as may be realized, such as using received radiation ofappropriate form, such as acoustical waves, optical radiation, infraredradiation, millimeter-wave radiation or other radio-frequency radiation,Terahertz radiation, and X-ray or Petahertz radiation. In someembodiments, the imaging apparatus may detect image information andproduce a detected signal. Those signals may then be acquired andprocessed to construct image data.

Additionally, or alternatively, one or more of the subject informationsources and/or the subject-associated item information sources mayinclude non-imaging sources. A non-imaging source may be a sensor orscreening apparatus that is configured to screen at least a portion ofthe subject and/or the subject-associated item(s), such as detectinggeneral features of the subject and/or subject-associated item(s), suchas a metal detector. The detection of the existence of a substance withthe person and/or the subject-associated item(s), an identification ofthe person, a characteristic, class, or categorization of the personand/or the subject-associated item(s), and/or other appropriateindicators or information may be considered features of the subjectand/or the subject-associated items. Alternatively, or additionally, thenon-imaging source may be context data stored in memory relating to thesubject and/or the subject-associated item(s).

The processed data from the subject information source(s) and thesubject-associated information source(s) may then be used to identifyfeatures at 460. Those features may then be correlated with each otherat 462. For example, the identification of an explosive substance on theperson from a screening apparatus may be correlated with the detectionof a triggering device in the person's carryon baggage and/or checkedbaggage.

The correlated features may then be classified at 464 based on thecombination of features. As discussed above, classification of featuresis a logical process for determining the likelihood that a detectedfeature is a suspect object or a false alarm, and the process may use arelational database to relate the various possible features relating toa subject and/or subject-associated item(s) that are surveilled. Oncethe features are classified, then conclusions may be generated at 466about the combination of features, and then outputted at 468. Theoutputs may be via display, report, and/or alarm condition.Additionally, or alternatively, the outputs may be used to controldownstream information sources, reanalyze other screening data, and/orstore data into memory. However, the steps discussed above may beperformed in different sequences and in different combinations, not allsteps being required for all embodiments of surveillance system 300.

A flow chart 500 illustrating another example of a method of operationof surveillance system 300, is shown in FIG. 23. Subject informationdata and/or subject-associated item information data from one or moreupstream information sources 502 may be used to operate one or moredownstream information sources of subject information data and/orsubject-associated item information data. Data from the upstreaminformation sources may be acquired at 504, and may be processed at 506.

The processed data from the upstream information source(s) may then beused to identify features at 508. Those features may be classified at510 based on the above described logical process and/or other suitablelogical process(es) for determining the likelihood that a detectedfeature is a suspect object or a false alarm. The identification offeatures and classification of features may be referred to as “analysisof the data.” Once the features are classified, then conclusions may begenerated at 512. Based, at least in part, on the generated conclusions,the controller may adjust the operation of one or more downstreaminformation sources, such as adjusting a sensitivity level or adjustingwhich portion(s) of the subject and/or subject-associated item(s) toscreen.

For example, a chemical substance may be detected in checked baggagescreening apparatus but may be present in a negligible amount. Thecontroller may increase the sensitivity level for the same chemicalsubstance in the subject and/or carryon baggage screening apparatusbased, at least in part, on detecting the chemical substance in thechecked baggage. Additional negligible amounts in the subject and/or thecarryon baggage, combined with the amount in the checked baggage, maycumulatively be non-negligible. However, the steps discussed above maybe performed in different sequences and in different combinations, notall steps being required for all embodiments of surveillance system 300.

A flow chart 550 illustrating another example of a method of operationof surveillance system 300, is shown in FIG. 24. Under the method, oneor more subject-associated items 314 may be screened at 552 via, forexample, at least one screening apparatus 26. The screening may includedetecting object information about an object potentially in the screenedsubject-associated item(s). The screening may include imaging at least aportion of the subject-associated item. Alternatively, or additionally,the screening may include detecting, in the subject-associated item(s),one or more of a metal substance, an explosive substance, a chemicalsubstance, and a feature identifying the person associated with thesubject-associated item.

Screening data may then be generated based, at least in part, onscreening at least a portion of the subject-associated item at 554. Thescreening data may include, at least in part, the object informationdetected. The generated screening data may be analyzed, for example bythe controller, at 556. Analyzing the screening data may includeassigning a relative weight, value, attribute, and/or relative indicatorto the object potentially in the screened subject-associated item based,at least in part, on the screening data. For example, a relative threatlevel (such as low risk, medium risk, or high risk) may be assigned tothe object potentially in the screened subject-associated item.

After analyzing the screening data, information data related to thesubject and/or one or more subject-associated items other than thescreened subject-associated item may be obtained at 558. Obtaininginformation data may include screening at least a portion of the subjectand/or the subject-associated item(s) other than the screenedsubject-associated item. That screening may include imaging at least aportion of the subject and/or the subject-associated item(s) other thanthe screened subject-associated item. Alternatively, or additionally,the screening may include detecting, in the subject and/or thesubject-associated item(s) other than the screened subject-associateditem, one or more of a metal substance, an explosive substance, achemical substance, and a feature identifying the person associated withthe subject-associated item. Alternatively, or additionally, theinformation data may include context data relating to the person and/orone or more subject-associated items associated with the person, otherthan the screened subject-associated item.

After obtaining the information data, the screening data may bereanalyzed at 560. The reanalysis may be based, at least in part, on theobtained information data, and may be in a manner that is at leastpartially different from the analysis of the screening data. Reanalyzingthe screening data may include assigning a relative weight, value,attribute, and/or relative indicator to the object potentially in thescreened subject-associated item based, at least in part, on theobtained information data. The relative weight, value, attribute, and/orrelative indicator assigned from the reanalysis may be the same, lower,or higher than the relative weight, value, attribute, and/or relativeindicator assigned from the analysis. For example, a knife potentiallyin the screened subject-associated item that was initially assigned ahigh threat level may be reassigned a lower threat level based, at leastin part, on obtained information data that the person sells knives andhas the appropriate approval to carry the knife in the screenedsubject-associated item. However, the steps discussed above may beperformed in different sequences and in different combinations, not allsteps being required for all embodiments of surveillance system 300.

A flow chart 600 illustrating another example of a method of operationof surveillance system 300, is shown in FIG. 25. Under the method,subject 32 may be screened at 602 via, for example, at least onescreening apparatus 26. The screening may include detecting objectinformation about an object potentially with or carried by the screenedsubject. The screening may include imaging at least a portion of thesubject. Alternatively, or additionally, the screening may includedetecting, in the subject, one or more of a metal substance, anexplosive substance, a chemical substance, and a feature identifying theperson.

Screening data may then be generated based, at least in part, onscreening at least a portion of the subject at 604. The screening datamay include, at least in part, the object information detected. Thegenerated screening data may be analyzed, for example by the controller,at 606. Analyzing the screening data may include assigning a relativeweight, value, attribute, and/or relative indicator to the objectpotentially with the screened subject based, at least in part, on thescreening data. For example, a relative threat level (such as low risk,medium risk, or high risk) may be assigned to the object with thescreened subject.

After analyzing the screening data, information data related to thesubject and/or one or more subject-associated items may be obtained at608. Obtaining information data may include screening at least a portionof the subject-associated item(s). That screening may include imaging atleast a portion of the subject-associated item(s). Alternatively, oradditionally, the screening may include detecting, in thesubject-associated item(s), one or more of a metal substance, anexplosive substance, a chemical substance, and a feature identifying theperson associated with the subject-associated item. Alternatively, oradditionally, the information data may include context data relating tothe person and/or one or more subject-associated items associated withthe person, other than the screened subject-associated item.

After obtaining the information data, the screening data may bereanalyzed at 610. The reanalysis may be based, at least in part, on theobtained information data, and may be in a manner that is at leastpartially different from the analysis of the screening data. Reanalyzingthe screening data may include assigning a relative weight, value,attribute, and/or relative indicator to the object potentially in thescreened subject-associated item based, at least in part, on theobtained information data. The relative weight, value, attribute, and/orrelative indicator assigned from the reanalysis may be the same, lower,or higher than the relative weight, value, attribute, and/or relativeindicator assigned from the analysis. For example, a gun potentially inthe screened subject that was initially assigned a high threat level maybe reassigned a lower threat level based, at least in part, on obtainedinformation data that identified the person as a police officer.However, the steps discussed above may be performed in differentsequences and in different combinations, not all steps being requiredfor all embodiments of surveillance system 300.

As shown in FIG. 21, some embodiments of surveillance system 300 mayrequire physical separation of the subject from subject-associateditem(s), screening of the subject, and/or subject-associated items whilephysically separate from each other, and returning those item(s) to thesubject. Surveillance system 300 may include any suitable structureconfigured to associate the subject and the subject-associated items,which may ensure proper return of those item(s) to the subject.

For example, as shown in FIGS. 26-27, surveillance system 300 mayinclude a plurality of tags 614 having at least one identifier 615. Thetags may be made from any suitable material(s), such as plastic, paper,leather, and/or metal. Tags 614 may include subject-associated item tags616 having at least one subject-associated item identifier 617, andsubject tags 618 having at least one subject identifier 620. In someembodiments of surveillance system 300, tags 614 may include onlysubject tags 614 or only subject-associated item tags 616.

Identifiers 615 may include any suitable identifier(s), such as one ormore of an image of the subject, a barcode, a code contained in an RFIDtag, a color, and an alphanumeric code. Where the tag is a RFID tag, thetag may be adapted to produce an identifying signal, such as when readby a RFID reader or RFID reading device. In some embodiments, the RFIDtag may include a memory 621 with the identifier stored in the memory.Although particular identifiers 615 are discussed, any suitableidentifier(s) may alternatively, or additionally, be used.

As discussed, tags 614 also may include at least one attachment assembly622. The attachment assembly may include any suitable assemblyconfigured to secure the tag to subject 32 and/or the subject-associateditems 314, such as one or more of ropes, cords, lanyards, pins, chains,etc. In some embodiment of tags 614, the attachment assembly may beexcluded. Although tags 614 are shown to be used to associate a personwith their subject-associated item(s), any suitable structure(s) mayalternatively, or additionally, be used, such as sticker(s), marker(s),etc.

One or more subject-related information sources 305 of surveillancesystem 300 may include at least one reader 624 configured to readidentifiers 620 of tags 614. For example, where the identifier includesa barcode, then reader 624 may include a barcode reader or barcodereading device. Additionally, or alternatively, where the identifierincludes an RFID device, then reader 624 may include an RFID reader. Thebarcode and/or RFID reader(s) may be connected and/or in communicationwith controller 302. The processor of the controller may associate theidentifiers and/or perform other suitable functions. Additionally, thememory of the controller may store the associations performed by theprocessor.

Additionally, or alternatively, surveillance system 300 may include oneor more containers 626 configured to contain subject-associated item(s),as shown in FIG. 27. The containers may include at least one base 628having at least one container identifier 630, which may include anysuitable identifier(s), such as one or more of an image of the subject,a barcode, an RFID device, a color, and an alphanumeric code.Additionally, identifier 630 may be read by reader 624. In someembodiments, containers 630 also may include a cover 632 and a lockingmechanism 634 configured to secure the cover to the base. The lockingmechanism may include any suitable mechanisms, including a mechanicalkey mechanism, a card reader, etc. Although containers 626 are shown tobe used to associate the person and their subject-associated item(s),any suitable structure(s) may alternatively, or additionally, be used.For example, a compartmentalized conveyor with identifier(s) for eachcompartment may be used.

A flow chart 650 illustrating an example of a method of operation ofsurveillance system 300 related to associating the person and the one ormore subject-associated items, such as carried items, is shown in FIG.28. A person and one or more carried items selected by the person to becarried by the person into a controlled access area may be associatedwith respective identifiers at 652. The carried item(s) may then bescreened while physically separated from the person at 654. Theidentifier(s) of the carried item(s) may then be matched with theidentifier(s) of the person at 656. If the identifiers match, thenpossession of the screened carried item(s) may be given to the person at658. In some embodiments, the person also may be screened whilephysically separated from the carried item(s) at 660. However, the stepsdiscussed above may be performed in different sequences and in differentcombinations, not all steps being required for all embodiments ofsurveillance system 300. Although the method is directed to associatingthe person and the carried items of the person, the method may beapplied to associating the person with any subject-associated item orany combination of subject-associated items that are physicallyseparated from the person.

Various embodiments of the method shown in flow chart 650 are possible,such as (a) carried item tag method, (b) carried item tag and subjecttag method, (c) container method, (d) container and subject tag method.In the carried item tag method, one or more subject-associated item tags616 having subject-associated item identifier(s) 617 may be attached (orbe required to be attached) to the carried item(s) as part ofassociating at 652. As discussed above, the subject-associated itemidentifiers may be one or more of an image of the person, a barcode, anRFID device, a color, an alphanumeric code, and/or other suitableidentifiers.

Where the subject-associated item identifier is an image of the person,associating at 652 also may include producing the at least onesubject-associated item tag with an image of the person (such as from animage taken of the person at a surveillance portal) and associating thattag with the one or more carried items. Matching identifiers at 656 maythen include comparing the image of the person on the at least onesubject-associated item tag to a visual appearance of the person. Wherethe tag is a RFID tag having memory 621, associating at 652 also mayinclude storing an identifier of the person, such as a name of theperson and/or other suitable identifiers, in the memory. Matchingidentifiers at 656 may then include reading memory 621 with a readingdevice and comparing the read identifier of the person with a documentthat identifies the person, such as an identity card and/or boardingpass.

Where the subject-associated item identifier is at least one barcode,associating at 652 also may include reading the barcode with a readingdevice and associating an identifier of the person with the barcode.Matching identifiers at 656 may then include reading the barcode withthe reading device and comparing the identifier of the person associatedwith the barcode with a document that identifies the person.

The carried item tag and subject tag method, similar to the carried itemtag method, also includes attaching or requiring attachment of at leastone subject-associated item tag 616 having at least onesubject-associated item identifier 617 (such as one or more of abarcode, an RFID device, a color, and an alphanumeric code) as part ofassociating at 652. However, unlike the previously discussed method,associating at 652 also includes providing, to the person, at least onesubject tag 618 having at least one subject identifier 620 (such as oneor more of a barcode, an RFID device, a color, and an alphanumericcode), reading the subject-associated item identifier(s) and the subjectidentifier(s), and associating or relating those two types ofidentifiers. For example, where subject-associated item identifier 617and subject identifier 620 include barcodes, associating at 652 mayinclude reading the barcodes with a reading device and associating thebarcodes with each other.

Matching at 656 may then include reading the subject-associated itemidentifier(s) and the subject identifier(s), and confirming that thesubject-associated item identifier(s) is associated with the subjectidentifier(s). In the example above where barcodes are used foridentifiers, matching at 656 may include reading the barcodes with areading device and confirming that the barcodes are associated.

In the container method, associating at 652 may include placing orrequiring placement of the carried item(s) in one or more containers 626having at least one container identifier 630. As discussed above,container identifier 630 may include one or more of an image of theperson, a barcode, an RFID device, a color, and an alphanumeric code.Where the container identifier is an image of the person, associating at652 also may include placing the image of the person on the container.Matching identifiers at 656 may then include comparing the image of theperson on the container to a visual appearance of the person.

Where the container identifier includes an RFID tag having memory 621,associating at 652 also may include storing an identifier of the person,such as a name of the person and/or other suitable identifiers, in thememory. Matching identifiers at 656 may then include reading memory 621with a reading device and comparing the read identifier of the personwith a document that identifies the person, such as an identity cardand/or boarding pass. Where the container identifier is at least onebarcode, associating at 652 also may include reading the barcode with areading device and associating an identifier of the person with thebarcode. Matching identifiers at 656 may then include reading thebarcode with the reading device and comparing the identifier of theperson associated with the barcode with a document that identifies theperson.

In some embodiments where container 626 includes cover 632 and lockingmechanism 634 configured to secure the cover, associating at 652 alsomay include securing the cover, such as when the carried items are inthe container and the container identifier has been associated with anidentifier of the person. Additionally, matching identifiers at 656 alsomay include opening the cover to allow removal of the carried item(s)from the container, such as when the container identifier and identifierof the person match.

The container and subject tag method, similar to the container method,may also include placing or requiring placement of the carried item(s)in one or more containers 626 having at least one container identifier630 (such as one or more of a barcode, a RFID device, a color, and analphanumeric code) as part of associating at 652. However, unlike thepreviously discussed method, associating at 652 may also includeproviding, to the person, at least one subject tag 618 having at leastone subject identifier 620 (such as one or more of a barcode, an RFIDdevice, a color, and an alphanumeric code), reading the containeridentifier(s) and the subject identifier(s), and associating or relatingthose two types of identifiers. For example, where container identifier617 and subject identifier 620 include barcodes, associating at 652 mayinclude reading the barcodes with a reading device and associating thebarcodes with each other.

Matching at 656 may then include reading the container identifier(s) andthe subject identifier(s), and confirming that the containeridentifier(s) is associated with the subject identifier(s). In theexample above where barcodes are used for identifiers, matching at 656may include reading the barcodes with a reading device and confirmingthat the barcodes are associated.

In some embodiments where container 626 includes cover 632 and a lockingmechanism to secure the cover, associating at 652 also may includesecuring the cover, such as when the carried items are in the containerand the container identifier has been associated with an identifier ofthe person. Additionally, matching identifiers at 656 also may includeopening the cover to allow removal of the carried item(s) from thecontainer, such as when the container identifier and identifier of theperson match. Although particular methods are discussed, any suitablemethod(s) may be used to associate the subject and the one or moresubject-associated items, such as carried items. For example, the atleast one identifier of the subject-associated item tag(s) may beassociated with at least one identifier of the person's ticket and/orboarding pass (such as a barcode of the subject-associated item and abarcode of the person's boarding pass).

An example of surveillance system 300 having a subject informationassembly 306 mounted in a surveillance station or portal 700 is shown inFIG. 29. Unless otherwise specified, surveillance portal 700 may includeat least some of the structures and/or functions of structures ofsurveillance portal 100 described above. The surveillance portal mayinclude an enclosure 702 and a plurality of screening apparatus 704. Theenclosure may at least substantially enclose subject 32 in subjectposition 34 and may include an entrance 704 and an exit 706. Theentrance and/or exit may be controlled barriers and may be controlled bycontroller 302. For example, controller 302 may open the entrance and/orexit only when screening apparatus 704 have completed screening thesubject.

Although enclosure 702 is shown to have entrance 704 and exit 706, theenclosure may alternatively have a single opening that serves as theentrance and the exit. Additionally, although enclosure 702 is shown tocompletely enclose subject 32, the enclosure may less than completelyenclose the subject. For example, the enclosure may exclude a top orceiling and/or may exclude a bottom or floor.

The plurality of screening apparatus may be adapted to detect on subject32 in subject position 34 one or more characteristics of at least oneobject potentially carried by the subject and/or the subject-associateditem(s). For example, the plurality of screening apparatus may include amillimeter wave imaging apparatus 710, a nuclear detection apparatus712, a biochemical detection apparatus 714, trace element detectionapparatus 716, a proximity card reader 718, and a subject recognitionapparatus 720.

Nuclear detection apparatus 712 may include any suitable structureadapted to detect the presence of one or more radioactive materials inthe subject and/or the subject-associated item(s). Examples of nucleardetection apparatus are provided in U.S. Pat. Nos. 7,142,625 to Jones etal. and 5,371,362 to Mestais et al., the complete disclosures of whichare herein incorporated by reference for all purposes. Biochemicaldetection apparatus 714 may include any suitable structure adapted todetect the presence of one or more biological and/or chemicalsubstances, such as biological and chemical warfare agents. Thebiochemical detection apparatus may include immunoassay systems,polymerase chain reaction units, liquid-phase systems, gas-phasesystems, flame spectrophotometry, infrared spectroscopy, and lasertechniques. Other examples of biochemical detection and/or nucleardetection apparatus are provided in U.S. Pat. Nos. 7,088,230; 7,084,753;and 7,061,388 to Maurer et al., the complete disclosures of which areherein incorporated by reference for all purposes.

Trace element detection apparatus 716 may include any suitable structureadapted to detect the presence of one or more substances in the subjectand/or the subject-associated item(s). The trace element detection maybe based on one or more technologies discussed above. Proximity cardreader 718 may include any suitable structure adapted to read and/oridentify one or more items, such as a boarding pass, an identity card,etc. The proximity card reader may include one or more barcode readers,radio frequency identification (RFID) readers, and smart card readers.Although proximity card reader 714 is included with portal 700, theportal may alternatively, or additionally, include other types of cardreaders, such as magnetic stripe readers, etc.

Person recognition apparatus 720 may include any suitable structureconfigured to identify the person in the portal. For example, the personrecognition apparatus may include any suitable technologies for faceimaging, retinal imaging, fingerprint and/or other biometricidentification of the person. Additionally, or alternatively, personrecognition apparatus 720 may include a video camera, and/or othersuitable camera.

Although portal 700 is shown to have specific types of screeningapparatus, the portal may have any suitable type(s) of screeningapparatus adapted to screen the subject in the subject position.Additionally, although portal 700 is shown to include six types ofscreening apparatus, the portal may have any suitable number ofscreening apparatus. Moreover, although portal 700 is shown to includethe subject information assembly, the portal may additionally, oralternatively, include the subject-associated item information assembly.

While the inventions defined in the following claims have beenparticularly shown and described with reference to the foregoingembodiments, those skilled in the art will understand that manyvariations may be made therein without departing from the spirit andscope of the inventions. Other combinations and sub-combinations offeatures, functions, elements and/or properties may be claimed throughamendment of the present claims or presentation of new claims in this ora related application. Such amended or new claims, whether they aredirected to different combinations or directed to the same combinations,whether different, broader, narrower or equal in scope to the originalclaims, are also regarded as included within the subject matter of thepresent disclosure.

The foregoing embodiments are illustrative, and no single feature orelement is essential to all possible combinations that may be claimed inthis or later applications. Where the claims recite “a” or “a first”element or the equivalent thereof, such claims should be understood toinclude one or more such elements, neither requiring nor excluding twoor more such elements. Further, cardinal indicators, such as first,second, and third, for identified elements are used to distinguishbetween the elements, and do not indicate a required or limited numberof such elements, nor does it indicate a particular position or order ofsuch elements unless otherwise specifically stated.

INDUSTRIAL APPLICABILITY

The methods and apparatus described in the present disclosure areapplicable to security, monitoring and other industries in whichsurveillance and imaging systems are utilized.

1. A surveillance method, comprising: screening by a surveillance systemapparatus at least a portion of a subject-associated item associatedwith and not including a person being screened, and not including anitem in the physical possession of the person during screening of theperson; generating by the surveillance system apparatus screening databased, at least in part, on the screening of at least a portion of thesubject-associated item; analyzing the screening data by a processor ofthe surveillance system apparatus; obtaining by the surveillance systemapparatus, after analyzing the screening data, information data derivedfrom at least one of (a) the person and (b) one or moresubject-associated items other than the screened subject-associateditem; and reanalyzing by the processor of the surveillance systemapparatus the screening data based, at least in part, on the informationdata, and in a manner that is at least partially different from theanalyzing of the screening data.
 2. The method of claim 1, in whichscreening at least a portion of a subject-associated item includesdetecting object information about an object potentially in the screenedsubject-associated item, and generating screening data includesgenerating screening data that includes, at least in part, the objectinformation.
 3. The method of claim 2, further comprising assigning oneor more of a relative weight, value, attribute, and relative indicatorto the object potentially in the screened subject-associated item based,at least in part, on the screening data.
 4. The method of claim 3,further comprising, after reanalyzing, reassigning one or more of arelative weight, value, attribute, and relative indicator to the objectpotentially in the screened subject-associated item based, at least inpart, on the information data.
 5. The method of claim 1, in whichscreening at least a portion of a subject-associated item includesimaging at least a portion of the subject-associated item.
 6. The methodof claim 5, in which imaging at least a portion of thesubject-associated item, includes transmitting, toward thesubject-associated item, electromagnetic radiation in a frequency rangeof about 30 PHz to about 30,000 PHz.
 7. The method of claim 1, in whichscreening at least a portion of a subject-associated item includesdetecting, in the subject-associated item, one or more of a metalsubstance, an explosive substance, a chemical substance, and a featureidentifying the person associated with the subject-associated item. 8.The method of claim 1, in which obtaining information data includesscreening at least a portion of the person and generating subjectscreening data, at least as part of the information data, based on thescreening at least a portion of the person.
 9. The method of claim 8, inwhich screening at least a portion of the person includes imaging atleast a portion of the person.
 10. The method of claim 9, in whichimaging at least a portion of the person includes transmitting towardand receiving from the person in a subject position, electromagneticradiation in a frequency range of about 100 MHz to about 2 THz, frompositions spaced from the subject position.
 11. The method of claim 8,in which screening at least a portion of the person includes detecting,on the person, one or more of a metal substance, an explosive substance,a chemical substance, and a feature identifying the person.
 12. Themethod of claim 1, in which obtaining information data includesscreening at least a portion of the one or more subject-associated itemsother than the screened subject-associated item, and generatingsubject-associated item screening data, at least as part of theinformation data, based on the screening at least a portion of the oneor more subject-associated items other than the screenedsubject-associated item.
 13. The method of claim 12, in which screeningat least a portion of the one or more subject-associated items otherthan the screened subject-associated item includes imaging at least aportion of the one or more subject-associated items other than thescreened subject-associated item.
 14. The method of claim 13, in whichimaging at least a portion of the one or more subject-associated itemsother than the screened subject-associated item includes transmitting,toward the one or more subject-associated items other than the screenedsubject-associated item, electromagnetic radiation in a frequency rangeof about 30 PHz to about 30,000 PHz.
 15. The method of claim 12, inwhich screening at least a portion of the one or more subject-associateditems other than the screened subject-associated item includesdetecting, in the one or more subject-associated items other than thescreened subject-associated item, one or more of a metal substance, anexplosive substance, a chemical substance, and a feature identifying theperson associated with the one or more subject-associated items.
 16. Asurveillance method, comprising: screening by a surveillance systemapparatus at least a portion of a person; generating by the surveillancesystem apparatus screening data based, at least in part, on thescreening of the at least a portion of the person; analyzing thescreening data by a processor of the surveillance system apparatus;obtaining by the surveillance system apparatus, after analyzing thescreening data, information data derived from one or moresubject-associated items other than the person and other than an item inthe physical possession of the person during screening of the person;and reanalyzing by the processor of the surveillance system apparatusthe screening data based, at least in part, on the information data, andin a manner that is at least partially different from the analyzing ofthe screening data.
 17. The method of claim 16, in which screening atleast a portion of a person includes detecting object information aboutan object potentially carried by the person, and generating screeningdata includes generating screening data that includes, at least in part,the object information.
 18. The method of claim 17, further comprisingassigning one or more of a relative weight, value, attribute, andrelative indicator to the object potentially carried by the personbased, at least in part, on the screening data.
 19. The method of claim18, further comprising reassigning one or more of a relative weight,value, attribute, and relative indicator to the object potentiallycarried by the person based, at least in part, on the information data.20. The method of claim 16, in which screening at least a portion of aperson includes imaging at least a portion of the person.
 21. The methodof claim 20, in which imaging at least a portion of the person includestransmitting toward and receiving from the person in a subject position,electromagnetic radiation in a frequency range of about 100 MHz to about2 THz, from positions spaced from the subject position.
 22. The methodof claim 16, in which screening at least a portion of the personincludes detecting, on the person, one or more of a metal substance, anexplosive substance, a chemical substance, and a feature identifying theperson.
 23. The method of claim 16, in which obtaining information dataincludes screening at least a portion of the one or moresubject-associated items and generating subject-associated itemscreening data, at least as part of the information data, based on thescreening at least a portion of the one or more subject-associateditems.
 24. The method of claim 23, in which screening at least a portionof the one or more subject-associated items includes imaging at least aportion of the one or more subject-associated items.
 25. The method ofclaim 24, in which imaging at least a portion of the one or moresubject-associated items includes transmitting, toward the one or moresubject-associated items, electromagnetic radiation in a frequency rangeof about 30 PHz to about 30,000 PHz.
 26. The method of claim 23, inwhich screening at least a portion of the one or more subject-associateditems includes detecting, in the one or more subject-associated items,one or more of a metal substance, an explosive substance, a chemicalsubstance, and a feature identifying the person associated with the oneor more subject-associated items.